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O ■ 1 Introduction 

(N : 

^ ■ Notion of character of representation plays an important role in the representa- 
Q ■ tion theory of groups. For finite dimensional representation a character x{9) is 
defined as a trace of matrix of operator Tg. This definition cannot be directly 
applied for infinite dimensional representations. For some representations of 
H ■ Lie groups one can define a character as a generalized function on the group as 
■ follows. 

+^ ! Extend a representation Tg of a group G to the representation of the group 
algebra L}{G) by the formula 



= y (f{g)Tgdg, 



> 

\0 where dg is the left-invariant measure on the group. Suppose that for any finite 



function the operator T^p has the trace. Then the formula 

. defines the trace xio) ^ generalized function on the Lie group G. 
^ It was proved in the paper |[T] that any irreducible representation of a con- 

nected nilpotent Lie group has a trace in mentioned sense. For any connected 
nilpotent Lie group and for a character of irreducible representation, which is 
associated with a coadjoint orbit Q, the formula of A.A.Kirillov is valid 

(X,^) = J (f{a)dnfi{a), (1) 

where 

if{a) = f (/?(exp(x))e2'^*^«'^)da: 
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is the Fourier transformation with respect to a given Lebesgue measure dx on 
the Lie algebra g of G, dfifi is the invariant measure on the orbit Q (see [T] and 
[21 chapter 3]). But it may be very difficult to use this formula for calculation 
of the character, when the orbit is a complicated manifold, defined by a system 
of many equations. It will be better to use the other approach: if we represent 
the operator T^p in the integral form 

T^F{x) = J A{x,y)F{y)dy, (2) 

then the trace is equal to J A{x^ x)dx (see [JLj). The (5-function type multipliers 
rise in the character; it implies that the support of character doesn't coincide 
with the group. Calculations of characters for different connected nilpotent Lie 
groups imply the following general conjecture. 

Conjecture 1.1. Given a connected nilpotent Lie group, the support of 
a character of irreducible representation, associated with an coadjoint orbit 
coincides with the closure of union of stabilizers ^ where f E 

The regular characters (i.e. characters of irreducible representations, asso- 
ciated with orbits of maximal dimension) of UT(n, M) were calculated in the 
paper [T]. Recall that any coadjoint orbit has even dimension. An coadjoint 
orbit is called subregular if its dimension equals to — 2, where c/ is a max- 
imal dimension of coadjoint orbits. A character of irreducible representation, 
associated with a subregular coadjoint orbit, is called a subregular character. 
Subregular coadjoint orbits of the group UT(n, E) were classified in [1]. Basing 
on this classification we obtain formulas (theorems 13.31 and 13.61 ) for subregular 
characters. In this paper we also calculate regular characters (see theorems 
12.11 and 12.21 ). since there is no proof of this formulas in [1] and there is some 
unexpected misprint (in the case of odd n). The obtained formulas confirm 
Conjecture 11.11 in the case of regular and subregular characters of the group 
UT(n,M) (see TheoremO). 

Note that subregular characters for the case of finite field were obtained in 
[3]. We use the following notations: 

1) if C is a matrix with functional entries, then 6{C) is a product of ^-functions 
at zero of its entries ; 

2) if C is an unitriangular matrix with functional entries upper the diagonal, 
then we preserve the notation 6{C) for the product of ^-functions at zero of its 
entries upper the diagonal; 

3) if Ci, . . . , Cm is a system of matrix, then 6{Ci, . . . , Cm) = ^{Ci) • • ■ S{Cm)- 
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2 Regular characters 

The group G = UT(n,IR) is a group of upper triangular matrices of size n 
with units on the diagonal. Its Lie algebra g = ut(n, M) consists of all upper 
triangular matrices with zeros on the diagonal. Applying the Killing form (•, •) 
we identify the conjugate space q* with the set of all lower triangular matrices 
with seros on the diagonal. According to the orbit method of A.A.Kirillov, 
there exists one to one correspondence between irreducible representations of 
a connected nilpotent Lie group and its coadjoint orbits. Any element / G 0* 
has a polarization p (i.e. a subalgebra that is a maximal izotropic subspace 
with respect to the skew symmetric bilinear form /([x, ?/])). The irreducible 
representation, associated with the coadjoint orbit is induced from one 

dimensional representation 



of the subgroup exp(p). 
2.1 Case of even n. 

Let n = 2k. Let us represent the general element g G UT(n, M) in the form of 
block matrix 



where are blocks of size k x k. Any regular coadjoint orbit of the group 
UT(n,M), where n = 2/c, contains a unique element of the form 



^(exp(a:)) 



^2nif{x)^ where x ep 




(3) 






3]). We use the following 



notations. For every 1 ^ s ^ /c we denote 



1) As = ; • • . ; the left 




Ck,l ■ ■ ■ Ck^k-s+l 



2) P^ = ^ 
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Theorem 2.1 (. p[]). Let n = 2k. The character of the irreducible rep- 
resentation, associated with the orbit of element / G ut(n,M)* of (Hj), has the 
form 

x(r/) = <5(Cn,C22)xl(Ci2), (5) 

where 

Proof. The subalgebra I ^ ^ 1 is a polarization of /. The irreducible rep- 
resentation, associated with the orbit r^(/), is induced from the one-dimensional 
representation e'^'^'^^^'^^ of subroup | ^ q ^ ^ | ' ^^^^ representation is realized 

in the space L^(A:'), where A! consists of matrices X = diag (Xn, X22) with 
blocks of UT(A;,R), by the formula 



TgF{X) = e^'''^^'^''^''^^''^''^'') F{XnCn: X22C22). 
Operator Tg is extended to operator 

T^F{X) = J (^(Cn,Ci2,C22)e2^<^'^"^-(^-^-)")F(XuCn,X22C22)c?CndCi2cia 
After substitution Yu = XuCu, Y22 = -^22(^22 we obtain in the form (EI): 

T^F{X) = J (^(Xf/yn,C'i2,X22V22)e'"<^'^"^-^-')F(yu,y22)c/yilrfl22C/Ci2. 

Then 

{X.<f) = J (/^(£;,Ci2,E)e2'^<^'^"^-^-)(iXnrfX22rfCi2. 
After substitution X22 by X22 we have 

(g) = 6{Cn, C22) J e2-(^'^"^-^-)ciXn(iX22 (6) 

Let Xii,X22 be unitriangular matrices of size k with entries x^j, ijij, 1 ^ 
i < j ^ k respectively. 

For every 1 ^ s ^ k and l^j^/c — s + lwe apply the notations 



X 



where dxg = dxs,s+i ■ ■ ■ dx^^k and dy^ = dyi^k-s+i ■ ■ ■ dyk^s,k^s+i- Taking into 
account ([7j), the formula (E]) is rewritten in the form 



x(^) = ^(Cn,C22)nx,s(^) (8) 

s=l 

Apply the well known equality 

in (Ej), we obtain 

Xs{9) = j • • • , Cs^k-s)e^^'^''-''-^^'dxs. (9) 

Let {x^g gj^i^ . . . , x^^) be a solution of the system of linear equations 

Cs,l = Cs^l + Xs,s+lCs+l,l + • • • + Xs,kCk,l = 0, 

(10) 

C-Sjk—s ^s^k—s ~l~ '^s,s+lCs+l,fc— s ~l~ • • • ~l~ X g j^Cf^ j^—g 0. 

Denote 

P's = Cg^k-s+l + x'i^s+l'^s+l,k-s+l + • • • + X^g^f^Ck^k-s+l- 

Let us introduce a new variable Xg^g. The vector (1, x^g^g^i, . . . , ^) is a solution 
of the system of linear equations 

Xs,sCs,l + Xs,s+lCs+l,l + . . . + Xg^kCk,! = 0, 



SjS^s^k—s ~l~ '^s,s+lCs+l,fc— s ~l~ • • • ~l~ X gj^Cj^ j^^g 0, 
X s^sC-s,k~-s+l ~\~ >^s,s+l'-'S+l,fc— s+1 ~l~ • • • ~l~ ^^SjfcC/Cjfc— s+1 Pg- 

Using the Cramer formulas in (JTTl). we have 



A. 

It implies 

Substituting Xs{g) into (Ej), we obtain the formula for the regular character for 
even n. □ 
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2.2 Case of odd n. 



Let n = 2A; + 1. Let us represent the general element g G UT(n, IR) in the form 
of block matrix 

/ Cii Ci2 Ci3 \ 

^ = 1 C723 , (12) 
V C33/ 

where the partition of matrix into blocks corresponds to the partition (/c, 1, k) 
of its rows and columns. Any regular coadjoint orbit of the group UT(n,]R), 
where n = 2/c + 1, contains a unique element of the form 












I 














/ 






/= , A= : ••. : , (13) 



where Ai, . . . , Aa; G (see [H §3]). 

Theorem 2.2 (. P). Let n = 2k + 1. The character of the irreducible 
representation, associated with the orbit of element / G ut(n,]R)* of (fT^. has 
the form 

X{g) = 6{Cn. C33, C12, C23) jl^, (14) 



where g as in (IT^. Xa(^13) 

/ * \ 

Proof. Subalgebra j * I is a polarization for /. The irreducible rep- 

V / 

resentation, associated with the orbit Q{f), is induced from the one-dimensional 

r/0 5i3\i 

representation e^'^*'^^'^") Qf subgroup < ^23 / • This representa- 

I vo y J 

tion is realized in the space L^(A'), where A:' consists of matrices 



^11 


Xi2 








1 











-^33 



X = 
by the formula 

TgF{X) = e2"<^'(^"^i3+^i^^^^)(^3^^^3)"')F(XnCn,XnCi3 + Xi2,X33C33] 
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Arguing as in subsection 2.1, we prove that 



Rewrite this formula in the form 



where 



Xig) = ^(Cii, C33, C12) J - J e2-(^'^-^^^^-)(iXn(iX33, (15) 

f g27ri(A,Xi2C23X33)^J^^2. 



Let us show that 



J = 



|Ai • • • Xk\ 



(16) 



Let 



X12 — 



C23 = (Cl, . . . , Cfc), X23 = 



V 



1 


yi2 ■ 


• yik 





1 . 


■ y2k 





. 


. 1 



Then 



(A,Xi2C23X33)=Xl'^^^^' 



where 



Ms = Xs,k+1 (ciyi^k-s+l + • • • + Ck-syk-s,k-s+l + Ck-s+l) ■ 

In particular, Mk = Xk,k+iCi, Mk-i = Xk-i,k+i {ciyu + C2). Denote 

(A, X12C23X33), = = y • • • dxk,k+i 



s=t 



Note that 

(A, Xi2C23X33)^ = (A, X12C23X33) , Ji = J, 

(A, -^12^*23-^33)^ = A^Mi + (A, Xi2C23-^33)t+i • 

We use induction on t, moving in decreasing order from /c to 1, to prove 

1 



Jt = S{ci, . . .,Ck-t+l)- 



Xf ■ ■ Xk\ 



(17) 



7 



For t = k we have 



this proves (TTTI). Assume that ( TTTI ) is proved for i + 1; let us prove it for t: 



t,k+l 



dx 



k,k+l 



^2mX,M,g^^^^ . . . , Ck-t)- L^dXtMl = 5{C1, • • • , Ck-t+l] 



1 



This proves (1T7|). Substitute ( fTTl ) into (JTHI). Calculation of the integral 
concludes similarly subsection 2.1. □ 

3 Subregular characters 
3.1 Case of even n. 

Let n = 2{k+m+2) . Partition rows and columns into blocks (m, 1, 1, /c, /c, 1, 1, m). 
The general element g G UT(n, M) can be written as a block matrix 



9 = 



/ 


Cii 






























\ 






12 L'13 <-'14 L-15 L-16 

1 (^23 C24 £"25 C26 

1 C34 C35 C36 

C44 C45 C46 

C55 C56 

1 







Cyj C18 \ 

C37 C'ss 

(^47 C48 

C57 C58 

^67 Ces 

1 C78 

^88 



(18) 



It follows from |3l §3] that any subregular coadjoint orbit of the group UT(n, M), 
where n = 2(/c + m + 2), contains a unique element of the form 



/ = 












































































































A2 

















71 


























72 








73 









00000/ 



(19) 



where Ai (resp. A2) is a quadratic matrix of the form ([Ij) of size m (resp. k\ 
All entries of the second diagonal of the matrix Ai and 71,72 do not equal to 
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zero. All entries of the second diagonal of the matrix A2, besides the last entry 
(see (S!)), do not equal to zero. 

Denote by £"45 a system of k matrix units that correspond to the places of 
second diagonal of the block C45 of the matrix g (see (fTSj)). Similarly, £"18 is 
a system of k matrix units that correspond to the places of second diagonal of 
the clock Cig. The following lemma is proved by direct calculation. 
Lemma 3.1. 1) Stabilizer q-^ in the Lie algebra q = ut(n,M) is a subspace 
spanned by 

Ea5, Eis, E2Q, E27, £37, 72^23 + IiEqi] (20) 

2) Stabilizer in the Lie group G = UT(n, R) equals to £" + 

Construct the following table of size 8x8. By symbol "X" we mark all 
places (a, h) such that Ea^b rises in 





• 




• 




• 




X 






X 






X 


X 










• 




• 


X 


• 










X 


















• 




• 














X 


















• 


















Tab 


.e ] 





For any entry c of the matrix ( [THl ) that is lying in one of the blocks, marked 
in the Table 1 by the symbol "•" or that is lying in the block Cis, we define a 
rational function c as follows. Denote $ = {(2,3), (4,5), (6,7)}. Let c be an 
entry of the block Cij and $(c) be a set of all pairs (a, 6) G $ such that a > i 
and b < j. Let A(c) be a minor of the matrix (1181 ) with rows and columns 
of $(c). Let A(c) be a minor of the matrix (fTSI). constructed by adding the 
row (resp. column) of the entry c to the system of rows (resp. columns) of the 
minor A(c). Denote 

c = ±A(c)-iA(c), (21) 

where the sign ± coincides with the sign of entry c in the minor A(c). For 
example, if m = A; = 1 (i.e. every block Cij is an entry Qj), then 



C12 — C12, C34 — C34, C56 — C56, C78 — C78, Ci4 



-c. 



23 



Cl3 Ci4 
C23 C23 
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C36 = 



.-1 
-45 



C38 - C^5'^67 



C35 
C45 

C35 
C45 





C36 
C46 

C37 
C47 

C67 



C58 = 



.-1 
-67 



C57 
C67 



C58 



C16 



Cl3 
C23 





Cl5 
C25 
C45 



C16 
C26 
C46 



C38 
C48 
C68 



CI8 = 



"'^23"^<^45^<^67^ 



Cl3 

C23 






Cl5 
C25 
C45 





Cl7 
C27 
C47 
C67 



CI8 
C28 
C48 



Denote: 

1) Sq is the set of entries lying upper the diagonal in the blocks Cn, C44, C55, Cgg; 

2) (Si is the set of all rational functions c, where c runs through all entries in 
the blocks that is marked in the Table 1 by the symbol "•" ; 

3) 5*2 = {C23C35 + C24C45, C45C57 + C^qCq-j, 71C23 — 72(^67}, 

4) 5 = {Sq, Si, S2}, 

5) d{g) = det C23 • det C45 • det Cqj. Note that according to partition into blocks 
(fTSj). the blocks C23, Cqj have size 1 and det C23 = C23, det Cqj = Cqj. 

The algebra MfG] of regular functions on the group G = UT(n, R) admits 
localization M.'[G] by the denominator system generated by d{g). Note that 
S C Let /' be an ideal in M.'[G] generated by S. The annihilator of the 

ideal /' is the subset in G' = {g ^ G : d{g) 7^ 0}. 

Proposition 3.2. Let / be of the form (JT^. Then the closure in G of the 
annihilator of the ideal /' coincides with the closure of the set Adc{G-^). 
Proof. The Lemma 13.11 implies that a function of S annihilate G^ . One can 
prove directly that S annihilate AdciGf). Then Ann/' D AdciG^) n G'. The 
ideal /' is prime and its dimension coincides with the dimension of the set 
AdciGf). □ 

Theorem 3.3. Let n = 2(A; + m + 2). The character of the irreducible rep- 
resentation, associated with the orbit of element / G ut(n,M)* of (HM, has the 
form 

X{g) = S{S) ■ xliCis) ■ xlic,,) ■ xoig) (22) 

where Xa2(^45) as in ([S), the matrix Cig is filled by entries c, where c is an 
corresponding entry of the block Gig (see (EI]), Xai(^i8) as in (E]), and 



Xo(^) = 



1 _^2^j(^Qo+73C67 



|7i72|^ • d{gy 



Qo = C23C37 + C24C47 + C25C57 + C26C67- 

Remark, since the element 71C23 — 72C'67 belongs to the set 5, one can substitute 
^ by ^ in the formula for Xoig)- 
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Proof scheme. 

Item 1. Consider the special case m = 0. Partition of rows and columns has 
the form (1, 1, k, k, 1, 1). The general element g E G = UT(n,M) and /o G Q* 
are written as block matrices 



g 







Vo 



Cl2 

1 








Cl3 

C33 






Cl4 

C24 
C34 
C44 







Ci5 

C25 
C45 

1 





C16 \ 

(^26 

C36 
C46 
C56 

1 



/ 



/o = 





















\ 














































A 













71 

















/ 





72 








73 






Consider the subgroup B of all matrices 



b = 



/ 1 61 
' ' 





\o 



12 


bi3 


614 


bi5 


bi6 


\ 


1 











he 





&33 


&34 





he 










644 





he 













1 


he 


/ 














1 



Let 7rA(6i) be the irreducible representation from subsection 2.1 of the unitri- 

angular group Bi of all matrices 61 = ( ^'^^ ] 

\ (J Ou J 

The irreducible representation T^", that corresponds to /o G 0*, is induced 
from the representation 



_,27ri(7i6i5+72&26+73&56) 



of the subgroup B. Direct calculations in spirit of subsection 2.1 lead to proof 
of formula ([22]) for special case m = 0. 

Item 2. Consider the special case m = 1. Partition of rows and columns 
has the form (1, 1, 1, k, /c, 1, 1, 1). The general element g E G = UT(n, M) and 
f E g* are written as block matrices 



/ 1 C12 Ci3 Ci4 Ci5 C16 Ci7 C18 \ 

1 C23 C24 C25 C26 C27 C28 

1 C34 C35 C36 C37 C38 

C44 C45 C46 C47 C48 

C55 C56 C57 C58 

1 C67 Cgs 

1 C78 

\oo 1/ 



/ = 



/ 










VAi 



71 



72 




\ 














A2 

73 

00000/ 



where Ai G 
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Consider the subgroup of all matrices 

/ 1 &15 6i6 6l7 hs \ 

1 623 hi 625 he hi he, 

1 hi 635 ^36 hi hs 

7 _ 644 645 he hi hs 

^* ~ 655 he b^i 

1 667 

1 

\000 1/ 

Let b'^ be a matrix obtained from by deleting the first row and the last 
column, be a group of all such matrices. Let T^^{b'^) be an irreducible 
representation of the group B[ as in Item 1. 

The irreducible representation TJ, that corresponds to / G g*, is induced 
from the representation 

of the subgroup B. Direct calculations in spirit of subsection 2.1 lead to proof 
of formula ([22D for special case m = 1. 

Item 3. Case of general m. Consider the normal subgroup B^ of all g G 
such that Cii = Cgg = E. The character Xm of irreducible representation 
of the subgroup B^, that corresponds to the restriction of / on Lie{Bm), is 
calculated similarly to Item 2. The character x is induced from the character 
Xm of subgroup Bm- Calculation of character is similar to subsection 2.1. □ 

3.2 Case of odd n. 

Let n = 2(/c+m+2)+l. Partition rows and columns in blocks (m, 1, 1, /c, 1, /c, 1, 1, 
The general element g G UT(?t,, M) can be written as a block matrix 



/ 


Cn 


C12 


Cl3 


Ci4 


Ci5 


Cie 


Cl7 


C18 


C'lg 


\ 





1 


C23 


C24 


C25 


C'26 


C27 


C28 


C29 










1 


C34 


C35 


C36 


C37 


C38 


C39 















C44 


C45 


C46 


C47 


C48 


C49 


















1 


C56 


C57 


C58 


C59 





















Gee 


Cer 


Ces 


Ceg 
























1 


C78 


C79 




\ 























1 


CsQ 


/ 


























C99 



(23) 
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Any subregular coadjoint orbit of the group UT(n, R), where n = 2(/c + m) + 5, 
contains a unique element of the form 



/ 



/ = 



















































































































































A2 




















71 





























72 











73 








Ai 



























\ 



(24) 



where Ai (resp. A2) is a matrix of form ([Ij) of size m (resp. k). All entries of 
second diagonals of the matrices Ai, A2 and 71, 72 do not equal to zero [H, §3]). 

The systems of matrix units £"46, Eiq is defined by blocks C46, C19 similarly 
as in subsection 3.1. 

Lemma 3.4. 1) Stabilizer q-^ in the Lie algebra q = ut(n, M) is a subspace 
spanned by 

Ea6, Eig, E27, E2S, E^8, 72£'23 + 7l^78; (25) 

2) Stabilizer in the Lie group G = UT(n, R) is equal to £" + q-^ . 

Construct the following table of size 9x9. By symbol "X" we mark all 
places (a, b) such that Ea^b rises in 





• 




• 


• 




• 




X 






X 








X 


X 










• 


• 




• 


X 


• 










• 


X 


















• 


• 




• 














• 




• 
















X 




















• 





















Table 2. 

Denote = {(2,3), (4,6), (7,8)}. For any entry c of the matrix ([IHD that 



is lying in one of the blocks, marked in the Table 2 by the symbol "•" , or that 
is lying in the block Cig, we define a rational function c as in ( 12T| ) changing $ 
by Denote: 

1) So is the set of entries lying upper the diagonal in the blocks Cn, C44, Gqq, C99; 

2) Si is the set of all rational functions c, where c runs through all entries in 
the blocks that is marked in the Table 2 by the symbol "•" ; 

3) 5*2 = {C23C36 + C24C'46, G^qGqs + C47C78, 71C23 — 72C'78}, 
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4) 5 — {^o, ^i, S2}, 

5) ^^1(5') = det C23 • det C^q ■ det Cyg. 

The algebra M[G] of regular function on the group G = UT(n, R) admits lo- 
calization W[G] by the denominator subset generated by di{g). As above denote 
by /' the ideal in M.'[G] generated by S; note that Ann(/') C {g ^ G : d{g) 7^ 0}. 
Proposition 3.5. Let / has the form IHM. Then the closure in G of the anni- 
hilator of the ideal /' coincides with the closure of the set AdciG-^). 
Proof. Similarly to Proposition 13. 2[ 

Theorem 3.6. Let n = 2{k + m) + 5. The character of the irreducible rep- 
resentation, associated with the orbit of element / G ut(n,M)* of (HM. has the 
form 

x{9) = S(S)-xUC\<:)-^^-xK9)- (26) 

Here Xa2(^46) as in (Ej), the matrix C19 filled by the entries c, where c is the 
corresponding entry of the block C19 (see (1?T1). xlj^i^ig) as in (Ej), and 

|7i72|^ • I71C23I • diig)"^ 

Ql = C23C38 + C24C48 + C25C58 + C26C68 + C27C78- 

Proof is similar to Theorem 13.31 

Theorem 3.7. Conjecture 11.11 is valid for regular and subregular characters of 
the group UT(n,E). 

Proof. Propositions 13.21 13.51 and Theorems 12.11 12.21 13.31 13.61 imply the proof. 
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